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(57) Abstract: A method for tracking and measuring volume, shape and surfaces of objects, such as logs, simultaneously. In series 
of load and unload operations along a procurement line, series of images are captured by e.g., CCD stereo cameras with sufficient 
spatial resolution capabilities. Simultaneously, the location of objects (logs) are registered by a GPS system aiding the tracking 
of the objects. The load and unload operations are typically performed by machineries such as harvesters, forwarders or trucks all 
equipped with a crane. On such machines digital cameras are mounted on jib arms and a computer system is attached. A GPS system 
is mounted on the machines too and the computer system is attached here as well. Thus the real time processing of stereo images 
can be accomplished and the volume, shape and surface of the 3D objects are computed simultaneously with their location in space. 
The resulting data are sent to a central database that keeps track of objects and their locations. These data are then again available 
for the following step in the procurement line as "a priory" information , facilitating the computation of the size, shape, surface and 
location of logs (objects) at the current load/unload operation. A chain of information provided by this invention will profoundly 
increase efficiency of any production chain subjec table to the presented method. 
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Josephine: 

METHOD AND SYSTEM FOR AUTOMATIC MEASUREMENT AND TRACKING OF LOGS, INDUSTRIAL 
WOOD AND BOARDS 



Description 
Field of invention 

2s [0001] This invention relates to measurement and tracking of objects by 
use of photogrammetric methods. The invention targets the situation where 
bundles of such objects are handled for being collected in larger bundles or 
for further processing of individual objects. During this action the individual 
objects of a small bundle will be visible to a large extend. In its simplicity 

30 the idea then is by photogrammetric methodologies to obtain information 
of a particular object from imagery of one or more small bundles where it 
appears. More specifically the three dimensional (3D) extend of an object 
and properties related to its surface such at texture and other significant 
characteristics are extracted. Capturing imagery while grabbing and releas- 

35 ing the small bundles and/or at successive points dining the handling gives 
the information necessary to keep track of the transportation path in time 
and space of each individual object. Thus the methodology in total provides 
measurements and tracking of the individual objects. 

[0002] Clearly the invention encompasses a wide range of possible applica- 
40 tions. A generic example is an industrial process where a commodity travels 
through several steps of subprocesses during the manufacturing. The speci- 
fication here is focused on a unique design for use in the forestry and wood 
processing industry. 

[0003] Dining the last decades forestry has undergone a substantial degree 
4s of mechanization. Tbday wood is harvested and processed at an increasing 
speed and the wood industries makes up large units. Further the machinery 
used applies highly developed technologies. As a standard felling machinery 
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and saws at sawmills are equipped with various computer technologies to 

aid software for optimization in each step of the procurement process. At 
so the same time there is a growing public awareness of the management of 

natural resources and the utilization of those resources that are harvested. 

It is generally agreed that future advances relating to wood processing are 

to be obtained from further optimizations of the wood procurement process. 

The present invention facilitates optimizations throughout wood procure- 
ss ment process by providing detailed information of the wood (objects) at apy 

stage where the system is applied. 

General section 

[0004] The present invention is a real-time computer vision and tracking 
system to automatically locate and measure the size and quality of individual 
eo hard and softwood logs, pieces of industrial wood, and boards, hereinafter 
logs. 

[0005] This system is to rationalize, and increase efficiency and measurement 
accuracy throughout the wood procurement process. Farther, the precise 
tracking of the logs allows for accurate documentation in e.g., pursuit of 

es wood certification. 

[0006] By a computer vision system is understood a single or a series of 
computer sensor systems with integrated GPS. The vision system locates and 
measures the logs at several stages of some wood procurement process. The 
results of one or more of these measurements axe to be used independently 

70 or in conjunction to give precise and unbiased estimates of position, size and 
quality of logs at any specific stage. 

[0007] By tracking is understood a record of the positions of a recognized log 
along the procurement process where the system is applied e.g., the positions 
of a specific log from felling to any stage along the procurement process where 
7s the system is no longer applied. 
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[0008] The system will satisfy different imme diate demands for information 
of various actors along the procurement process. In particular: i) buyers 
and sellers demand for accurately measured and classified trading units and 
documentation hereof, and ii) the need to back-track logs and other wood 

so commodities to their origin in the forest for certification purposes. 

[0009] Further, the information collected by the system up to any given 
stage (operation) of the procurement process can be utilized for optimization 
purposes in subsequent stages of the procurement process. This increases the 
efficiency of the procurement process. 

8s [0010] A schematic representation of the wood procurement process is shown 
in Figure 1. The action pattern of logs being loaded and unloaded by similar 
devices e.g., cranes, is repeated at all operations throughout the procurement 
process. Therefore the system is designed to apply similar computer sensor 
systems and similar image analysis algorithms at all stages. 

so [0011] Typically logs are traded at one or more of the operations listed 
in Figure 1. Thus the unit for trade is a varying collection of logs and the 
system is designed to compute aggregate values for such collections of logs. 
[0012] The system can be applied at any action or set of actions e.g., an 
operation listed in Figure 1. In consequence it is the responsibility of buyers 

ss and sellers to decide on what stages it is appropriate to apply the system. 
Back-tracking boards to their origin in the forest does naturally require the 
system being applied at most unload and load points (actions). 
[0013] The present invention suggests measuring individual logs using a 
stereo vision system integrated with GPS. The GPS system provides the lo- 

ioo cation in space of the equipment that handles the logs and the vision system. 
Combined with automatic recognition of the logs in imagery captured by 
the vision system the path of each individual log can be mapped to provide 
tracking of the logs. 

[0014] The core idea of the vision system is that it captures imagery of 
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105 the logs during load/unload actions. During these actions only a few logs 
are handled at a time by a crane or similar device. Imagery of a bundle 
of few logs allows for a complete recognition of each individual log in the 
bundle. Candidate mount points of vision system are e.g., grab on harvester 
or forwarder, body of harvester or forwarder, truck at plant, conveyer belt 

no or any other place where the logs can be seen from vision system. 

[0015] Further, imagery of the stacks (collections of logs) that are being 
loaded or unloaded are captured continuously to monitor exactly where each 
bundle of logs are placed or taken from. In this way stacks are considered 
cohorts of individual logs with their individual properties such as transporta- 

n5 tion path and size. Thus aggregate quantities and distributional statistics at 
the stack level can be computed to characterize a stack. 
[0016] The idea of measuring logs on the fly by remote sensing and stacks 
being cohorts of logs with an associated record of information at the log level 
is an advance compared to todays practices. Common practices is to measure 

120 stacks in a separate process and characterize stacks by their outer measure 
and other aggregate quantities. 

[0017] Each individual log is being monitored several times during the 
procurement line. At any load/unload action stereo imagery of each log while 
located in the unload stack, the crane and the load stack is being captured. 

125 This comprehensive information source forms the basis of an image analysis 
algorithm to estimate the size and location of the logs with a high level of 
accuracy. Integrating imagery across several operations in Figure 1 in the 
image analysis adds to the level of accuracy that can be obtained. 
[0018] The image analysis algorithm is designed to adopt information from 

i3o many sources. Hence the algorithm allows data fusion from sources such 
as field inventory, harvester, regional prior information on size and quality, 
and information dynamically gathered during a forest operation to ensure 
self-calibration. 
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[0019] An important by-product of the real-time implementation of the 
las system is that the influence of decay and other biological factors can be 
quantified. Especially if there is a time-lag between the creation of a stack 
and it being picked up, using the system at both of these operations makes 
it possible to monitor changes in the stack. 

[0020] The present invention is considered applicable for other industrial 
uq uses where 3D objects are being handled in a similar repeated load/unload 
fashion in stacks. 

Specific section 

[0021] The system is designed to be build from standard components that 
complies with industry standards to ensure portability and low cost. In 
145 particular standard cameras, standard image formats, standard graphics hard 
and software, standard computer units with accompanying operating systems 
suffice to build an operational system. 

[0022] One example of an actual implementation of the this system is based 
on CCD digital cameras with a spatial resolution of 2-5 mm 2 per pixel for 
iso objects located at 2-15 m distance in physical space from the vision system. 
The image format is compressed tiff and the software implementation of this 
system written in C/C++/C# runs a windows or Linux base. The graphics 
procedures runs through an accelerated 3D graphics card using the OpenGL 
language. 

155 [0023] In the following is by machinery understood some machinery that 
handles logs along the wood procurement process. The device used by such 
machinery to handle the logs will be referred to as a grab. 
[0024] Some examples of machinery are Valmet and Ponsse harvesters 
and forwarders, Mack and Volvo trucks with trailers build to load logs and 

wo equipped with jibs, and genuine cranes at the saw or paper mills. 

[0025] The stereo vision system is to be mounted on the machinery such 
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that a continuous series of images of the logs that are being handled can be 
captured. The ideal strategy is to capture one or more set of images of the 
load and unload stacks and of the bundle of logs in the grab for each bundle 

165 of logs being handled. 

[0026] As an example harvesters have lights mounted pairwise on their jibs 
arms and shielded by a frame. A set of cameras are conveniently mounted 
next to the lights within these frames. When the operator fells a tree and at 
each time a log is cut off the vision system takes images. For a forwarder the 

i7o cameras are mounted similarly, but here the images are taken whenever a new 
cohort of logs i grabbed or released and at regular time intervals in-between 
or whenever the orientation of the cohort in the grab is optimal relative to 
the cameras. 

[0027] The GPS unit is mounted on the machinery too. From its continuous 
its logging of signals the absolute orientation and location in space of the ma- 
chinery is known. Given the orientation of the grab relative to the machinery 
the location in space of the grab is computed. 

[0028] Logs are three-dimensional (3D) solid objects. In consequence the 
image analysis algorithms are implemented by use of software routines sup- 
180 ported by standard 3D graphics hardware. Computing images of a virtual 
3D universe that represents the real world stacks and bundles of 3D logs then 
is fast and straight forward. 

[0029] The resulting information about stacks or bundle of logs computed 
from the images are subsequently stored in a central database. This informs^ 
185 tion include basicly estimates on size, location and quality of the individual 
logs. Further, aggregate values at stack level as well as image scenes (movie 
pictures) of part of or of the whole recorded operation should be stored in 
the database too. 

[0030] The central database can be located on any pre-selected machinery 
wo or office computer. The only requirement is that the other computers are 
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able to access the central computer by an online connection e.g., the Internet 
by a phone card or a similar standard device. This way the system is robust 
against hardware failure on any link in the procurement line. 
[0031] The image analysis algorithm for tracking and size estimation is 
Ms conceptually split into two stages. Stage 1 is a coarse recognition of the 
logs taken from or added to a stack, and the logs held in the grab. Stage 

1 is accomplished by template matching or other filtering and provides ap- 
proximate location and size of logs in the virtual 3D world. Stage 2 is to 
infer accurately about each individual log from the image data available and 

2qo other prior information. The approximate configuration of the virtual 3D 
world from Stage 1 is used as initial value for stage 2. Stage 2 is more in- 
volved and applies a formal statistical analysis. The outcome of the Stage 

2 are configurations of the virtual 3D world (one for each captured image 
used in the analysis) that describes the real world best given the information 

205 available. In other words, Stage 2 provides the transportation paths of the 
logs and their size and quality. 

[0032] Both of Stage 1 and Stage 2 require a specification of the virtual 
3D world that mimics the real world. That is, a mathematical model is set 
up to describe the 3D shape of the logs and their transportation path. This 
210 model is then adjusted (estimated) to fit best possible with the actual taken 
images. 

[0033] A possible mathematical model for a log is that the stem center fol- 
lows a 2nd, 3rd or 4th order polynomial (in a plane) and that cross-sections 
perpendicular to the stem center are circular with diameter a linear func- 
215 tion of distance to the stem base (Figure 4). Alternatively diameter can be 
specified by stem taper functions commonly described in literature. 
[0034] Given a specification of the virtual world the mapping into image 
space (image formation) is required to compute the images of the virtual 
world. It is by comparison of these computed images and the actual taken 
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220 images that the mathematical model can be estimated. 

[0035] As an example if the vision system is based on CCD cameras the 
mapping from object space (the physical world) into image space is the com- 
position of a central projection and the deterioration by the lens and irregu- 
larities induced by the chip. The two latter together makes up the so-called 

225 inner orientation and is a specific property of each camera. The inner orienta- 
tion is established separately from a test scene in a laboratory. The location 
and attitude of the cameras in the vision system relative to one-another to- 
gether with the model for image formation makes up a stereo vision system. 
That is, a system that allows for 3D reconstruction of the objects of interest 

230 (the logs). 

[0036] The Stage 1 filter that singles out individual logs in the grab operates 
on a complex source of information including one or more of: grab "width", 
location of grab in space (implicitly orientation of logs), exact record of the 
logs handled if taken from stack already monitored by the system, other prior 

23s information on log size e.g., from harvester. 

[0037] The Stage 1 filter that singles out individual logs taken or added 
to a stack operates on successive images of the stack to detect changes in 
its surface. Candidate locations of logs axe identified from these changes 
(Figure 3). Information about the location of the grab where it took or 

240 added logs and the bundle that was handled is used in support of this filter. 
[0038] In essence the Stage 1 filters map the complete transportation path 
of each individual log and the tracking is completed. Note that image data 
and GPS data are used in conjunction to complete this task. 
[0039] The inference framework for Stage 2 is Bayesian where the like- 

245 lihood term /(/; 6) is the density of the image data I under a statistical 
model parameterized by 6 and 7r(6) is the prior on 0. The statistical model 
encompasses the model of the virtual 3D universe, the image formation pro- 
cess and the randomness of noise in the imagery. The posterior p is given 
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by p{9\I) oc n(0)f(I;Q). Since 0 includes the parameterization of the vir- 
2so tual 3D world, maximizing p provides an estimate of the configuration of the 
3D real-world logs. A natural choice of the prior 7r is knowledge about the 
size distribution of the logs under study. Typically this information is easily 
obtained from growth models or yield tables. 

[0040] Technically each log in the virtual world is represented by a discrete 
255 spatial object during the maximization of the posterior. The set of points 
that spans the log are derived from the underlying mathematical model. The 
triangulation between these points then makes up the surface of the log. Let 
ff be the parameter that parameterizes a particular log. Then 0' is a subset 
of 0. Having maximized p with respect to 9 therefore provides the best value 
260 for 0'. Volume and curvature and other important properties of the log are 
therefore best computed from the value of & \ 

[0041] Since logs are solid objects with a certain regularity in shape, the 
parameter 9' can not be considered a completely free parameter. Thus the 
posterior p is maximized under suitable smoothness constraints on the shape 

265 of logs. Further different logs are in principle not allowed to occupy the same 
physical space. In practice this constraint is relaxed a bit to accommodate 
for the fact that 9 f is not a perfect representation of the real-world logs. 
[0042] If the system is applied at several stages along the wood procurement 
line, the image data / is the aggregate image data from all the consecutive 

270 images captured. This implies that the 3D configuration of each log is esti- 
mated from image data captured from different directions relative to the log 
minimizing the occluded volume of the logs. 

[0043] The system operates in absolute units in the sense that it produces 
the measured sizes in meters or some other absolute unit. Technically this 
275 requires the scale be known in the imagery captured by the stereo vision 
system. In recognition of the fact that the system operates in a disturbed 
environment, marks on the grab or other parts of the machinery is used in 
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addition to a known base line of the vision system to obtain scale. 
[0044] The Stage 2 image analysis gives a coarse assessment of wood quality 
bo e.g., based on curvature of a log and the appearance of growth rings at cross 
sectional cuts of logs. Additional image analysis routines are applied to 
measure rot percent, knots, bark texture changes to get as much information 
about wood quality as possible. 

[0045] When a stack of logs is completed the system computes aggregate 
aas quantities for the stack. The set of quantities computed may be changed 
dynamically by the user. Typical aggregate quantities are: number of logs 
in the stack, total volume of the logs in the stack, and size and quality 
distribution of the logs in the stack. 

[0046] Whenever logs are collected on some machinery and transported 

290 to some other location for unload the real-time requirement on the image 
analysis algorithms can be relaxed. It is not until the unload action takes 
place that the system must provide the image analysis results. This fact 
is particularly useful in the first stage where the system is applied is on a 
forwarder in the forest. At the first stage no prior information about the log 

295 is available and successful log recognition and size estimation may require 
an extended computation time. Usage of the system at subsequent stages 
along the procurement process will require less computer run time because 
detailed prior information about the logs are available. 
[0047] The vision system is mounted on the machinery so that it can capture 

300 images of both the grab and the load and unload stacks. The ideal situation 
is that the system be mounted such that for each bundle of logs that is being 
handled images of both grab and stacks can be captured. Some candidate 
mount points are jib arm or the body of the machinery as shown in Figure 2. 
[0048] A unique feature of the system is that the measuring device, i.e. the 

305 vision system, is not in contact with the logs. This implies that the system is 
less prone to deterioration, and thus the demand for continuous calibration 
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of the system while being applied is low. This is an advantage to measuring 

devices that are mounted in the felling device of a harvester. 

[0049] The present invention suggests both the tracking and the size estima- 

310 tion tasks solved by the system. As time progresses and e.g. bio-informatics 
technologies evolve, the tracking task may be completed from the DNA foot- 
print of each log or some other recognition procedure. By virtue the system 
is designed to incorporate such information in the Stage 2 image analysis. In 
other words, the system is designed to be complemented by other information 

sis sources that aid the tracking and size estimation. 

[0050] The most critical factors to the system are the sensor conditions and 
the number of logs that the grab handles. Operating the system during e.g., 
night time with a sensor sensible to visible light thus requires the logs being 
lighted by some artificial light source. If a particular log is occluded by other 

320 logs when located in the stack or in the grab, the system can only provide an 
approximate estimate of its size. The solution space for the volume occupied 
by that particular log is however quite small so that the overall performance 
of the system i not adversely effected. 

[0051] If the order of logs in a stack is modified by some outside factor or 
32s its geometric constitution has changed much, the change is recognized by the 
system and it starts monitoring if logs are missing in the stack. 
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Claims 

1. A method for tracking and determining volume, shape and surfaces of 
objects simultaneously, comprising the steps of 

aao - capturing an electronic image or series of images of objects 

- simustainiously locating objects in the real 3D world 

- electronically processing the images to obtain a set of data repre- 
senting the size, shape, surface and location of the objects in the 
real 3D world 

ass 2. A system for tracking and determining volume, shapes and surfaces of 
objects simultaneously, comprising 

- a camera system for capturing an electronic image of the stack 

- a computer system operatively connected to or incorporated in 
the camera system 

340 - a data interface for transmitting the image or data derived from 

the image to the computer system 

- the computer system being programmed to electronically process 
the image to obtain a set of data such as a complete 3D reconstruc- 
tion representative of the volume, shape and surface of individual 

345 logs 

- the computer system being programmed to electronically process 
the image to obtain a set of data representative of the volume of 
the stack. 

3, A system according to claim 2, further comprising 

350 - a GPS system for capturing the location and direction of the load 

and unload device. 
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4. A system according to claim 2 or 3, further comprising 

- a computer system operatively connected to or incorporated in 
the GPS system. 

355 5. A system according to claim 2-4, further comprising 

- a data interface for transmitting the GPS data derived to the 
computer system. 

6. A system according to any of the claims 2-5, wherein 

- the computer system being programmed to electronically process 
360 the data to obtain a set of location data of the individual logs. 
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Figure 1: Conceptual chart of wood procurement process. 



SUBSTITUTE SHEET (RULE 26) 



_ ifee'd^CT/pfo 'ifM ?o& 

2/3 




Figure 2: Schematic representation of logs in grab and location of sensor 
system. 




Figure 3: Schematic representation of a stack changing size. 
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